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ABSTRACT 

This synthesis paper represents the input of 60 
respondents to a working paper on a conceptual model of educational 
outcomes for children and youth with disabilities. The input is 
organized around the broad areas of: (1) underlying assumptions, (2) 
the conceptual outcomes model and indicators, (3) unresolved issues, 
and (4) general comments /concerns . Each of these sections includes a 
synthesis of comments that reflect support for a particular position, 
concerns, suggested alternatives or refinements, and sample comments. 
Comments related to the specific assumptions underlying the 
conceptual model of educational outcomes are summarized first, with 
one assumption about the model of outcomes, three about the 
indicators used to measure outcomes, and three about the 
comprehensive system of indicators. The section on responses to the 
conceptual models and indicators summarizes preferences for the two 
outcomes models proposed and proposes new models or new components of 
the proposed models. The section on unresolved issues contains 
comments about intended versus unintended outcomes, direct versus 
indirect outcomes, using the same versus different indicators for 
those with and without disabilities, using category specific versus 
noncategorical indicators, differentiating by severity of disability 
or by developmental level, and focusing on system level versus 
individual level indicators. In the final sect'^^, general comments 
are clustered under key areas that include the overall purpose of the 
model, the broader context within which outcomes are nested, the 
goals of education for those with and without disabilities, the tone 
and term^'nology used in the paper, and suggestions to address 
"missing ^^ieces." (JDD) 
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A Conceptual Model of Educational Outcomes for 
Children and Youth with Disabilities: 

NCEO Working Paper 1 



Overview of Woridng Paper 1 



Working Paper 1 . A Conceptual Model of Educational Outcomes for Children and Youth with 
DisatHlities. was initially distributed t>y staff of the National Center on Educational Outcomes in July 
of 1991 to approximately 175 individuals, requesting their input on the ideas presenteu therein. 
At the time of distritxjtion, the paper already reflected the input of more than 25 individuals who 
had met with Center personnel on one or more occasions to discuss the issues surrounding the 
development of a model of outcomes and a system of outcome Indicators for students with 
disabilities. Those providing early formative input on the initial draft included NCEO staff and 
consultants, state directors of special education from Kentucky, Utah, Massachusetts, South 
Dakota, Minnesota, Calrfomia and Connectk:ut, representatives fr^m OSERS, and members of 
the Mid-South RRC evaluatton forum (see attachment 1). This synthesis paper represents the 
input of over 60 additional respondents. 

The conceptual model paper is being revised on the basis of the input that is summarized in this 
document as well as on the basis of feedback from various groups with a stake or vested interest 
in the devetopment of a conceptual nrxxlel of outcomes and indtoators who have met with NCEO 
staff. The revised conceptual rrxxlel paper will serve as the foundatton for the subsequent 
modificatton ar>d expanston of a conceptual model and system of indtoators, whk^ is one of the 
major undertakings of the Nattonal Center on Educational Outcomes. 

An overview of the conceptual nfK)del paper (Working Paper 1} is provMed here to remind the 
reader of its structure. 

Introduction 

Informatton is provkJed on the need to examine educattonal outcomes for students with 
disabilities. Addittonal sections discuss the NCEO approach to kient'ifying outcomes. 

Definitions of Key Terms 

Three key terms (outcomes, indtoators, comprehensive system of irKlk:ators) are defined, 
atong with the key terms used within the def inittons. 

Fundamental Assumptions 

Seven assun4>tk)ns that underlie the NCEO approach are klentXied. One assumption 
relates to the nxKlel of outcomes, three to the outcome indtoators, and three to the 
comprehensive system of indk^ators. 

Preliminary Model of Outcomes 

Two alternative models are presented as potential models of outcomes. 

Illustrative System of Indicators 

A partial listing of possible indtoators is provkled. 

Unresolved Issues 

Seven unresolved issues that need to be addressed in conjunctton with 
selecting a model and a system of indk^ators are listed and briefly discussed. 
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Summary 



This synthesis is based on input from over 60 individuals who were not constrained in any way in 
giving their responses. Some sent responses to a list of questions, some sent letters, some sent 
specific comments on the paper itself, and some called in responses. Individuals sometimes 
focused on different aspects of the paper. Some responded to all or most areas of the working 
paper; others responded to just one issue or area. 

In synthesizing the responses, an attempt was made to best reflect as accurately as possible the 
range of responses as well as the frequency of responses. In this synthesis we summarize 
comments related to the Assumptions, Definitions, Models, and Unresolved Issues first, then 
present some general comments. This document represents a synthesis that includes trends in 
responses and sample comments from those who addressed particular areas or issues. 

The responses contain many statements of positive support for the efforts and cunrent status of 
NCEO*s work to date. The comments reflect both an appreciatton of the thoughtfulness and logic 
reflected in the working paper and of the comprehensive nature and importance of the task that it 
represents. The responses also contain some very thoughtful and detailed suggesttons for 
revisions that reflect the time and commitment devoted to this process by readers of NCEO's 
working paper. NCEO staff gratefully acknowledge the input of the indivkiuals who are listed on 
the to\hwivQ pages. 
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Introduction 

The inpiJt on "Working Paper V (NCEO, July, 1991) that is synthesized in this report is organized 
around the broad areas of underlying assumptions, NCEO's conceptual model and indicators, 
unresolved issues, and general comments/concerns. Each of these sections includes a 
synthesis of comments that reflect support for a particular position, concerns, suggested 
alternatives or refinements and sample comments. The comments that are included are 
presented as representative of the range of comments that were received and are not reported as 
quotations directly attributal)le to a particular respondent. While there is value in acknowledging 
the source of particular perspectives regarding an issue, it was beyorxl the scope of this report to 
obtain pentiission from the respondents to attribute comments to them. In September of 1992. 
NCEO's revised verston of the working paper ("Working Paper 2"), based on feedback from 
individuals, agencies and work groups as well as on the comments summarized in this report, will 
be available. 



Assumptions. Comments related to the specifk: assumpttons underiying NCEO's efforts to 
develop a conceptual model of educattonal outcomes are summarized first. Seven assumpttons 
underiying the NCEO approach were klentified in "Worthing Paper 1 ." One assumption was about 
the model of outcomes, three were about the indk^ators used to measure outcomes and three 
about the comprehensive system of indicators. A small number of indivMuals commented on the 
assumptions in general; more reacted to all or most of the specif ic assumpttons, provkling critique 
and comments on partk^ular issues of concern. 



Conceptual Model and indicators. The section on responses to conceptual oKKlels and 
indicators summarizes preferences for the two outcomes models that were proposed in "Woricing 
Paper 1 ." In addition to reacting to the nnodels as presented, several indivkJuals proposed new 
models or suggested components of models that could be incorporated into one of the models 
proposed by NCEO. Of the areas and issues to which readers were to respond, the largest 
number addressed the issue about the outcomes nfK>del(s). Two thirds of those providing 
feedback to the working paper commented on the outcome models. 



Unresolved Issues. The next section summarizes specify responses to unresolved issues 
that were raised in the wortcing paper. The unresolved issues that were raised included issues 
about on intended versus unintended outcomes, direct versus indirect outcomes, using the 
same versus different indicators for those with and without disabilities, using category specific 
versus noncategorical indteators, different;ating by severity of disatrility or by devetopmental level 
and focusing on system feva/ versus individual level indk^ators. In general, those who responded 
to one issue addressed each of the seven issues that were raised. HaH of those who responded 
to the woricing paper responded to the unresolved issues sectton. 



Genaral ComnMnts. Comments and concerns of a more general nature are summarized in 
the final sectton of this report. Many respondents addressed the questton of whether and how 
consensus couU be achieved in response to a specifk: question regarding the consensus issue. 
Many more raised issues and general concerns of interest to them. These general responses, to 
the extent that they clustered around themes, are synthesized under key areas that include: the 
overall purpose of the conceptual rrxxiel and system of indtoators, the broader context within 
whk;h outcomes are nested, the goals of educatk>n for those with and without disabilities, the 
tone and terminok)gy used in the paper and suggesttons to address what readers felt were 
"missing pieces." 
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Fundamental Assumptions 



The fundamental assumptions articulate NCEO*s perspec ive on special education as a 
capacity-txiilding resource for general education and a support to students and families. 
Special education is viewed as necessary to prevent impainnents from t)ecoming disabilities 
and for the purpose of compensating for disabilities. While schools today include students with 
a variety of disabilities, policy makers identifying goals and assessing outcomes for students in 
America's public schools often fail to consider students with disabilities. The fundamental 
assumi^ions or pren^ses that ^ide NCEO*s activities and t( which readers needed to respond 
relate either to the model of outcomes, to the specific outcomes indicators or to the 
comprehensive system of indicators. 

Briefly, the assumptions state that 

• the rTKx!el of outcomes should be comprehensive and apply to a// students, 

• the indicators of outcomes for students with and without disabilities should be related, 

• the indicators should respect the diverse characteristics of the school population, 

• indicator data may need to be included that is less than optimal by research standards, 

• a system of indicators should provide data for state and national policy decisions, 

• a system of indicators may include valued indicators even if functional relationships have 
not been established (as desired) between outcomes indicators and educational inputs, 
contexts and processes 

• a system c^ indicators needs to be flexible and responsive to change. 



Testimony on Fundamental Assumptions 
Number Responding with General Comments: 17 



Support for Fundamental Assumptions 

Of the 17 individuals who responded with general comments conceming the assumptions overall, 
three quarters of them stated that tney fett the assurnptions were appropriate and were an 
important part of the outcomes paper and process, while one suggested either expanding on 
them or deleting them. 



Concerns Regarding General Assumptions 

The expressed concerns included the following: 

• Making sure the assumptions are inclusive enough such that at a moimum they meet 
the standards for goal achievement covered in various curriculum guidelines. 

• Getting educators to make the necessary decisions to make the assumptions 
(simultaneously) work. 

• Determining how the stmctures in the models that specify outcome areas materialize in 
empirical data. 
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Suggested Alternatives or Refinements 

Two of the suggested alternatives were related to the characteristics of students: 

• Recognizing that the differing goals students bring to school may affect the 
"interactrons** under outconies. Acknowledge that the assumption that student goals 
do not differ is not so. Take motivatton and willingness to accept instruction into 
account. 

• Addressing the heterogeneity of the school population and the spillover of effects 
from other environments that are refiocted in educational outcomes. Avokl t>laming 
schools for things they cannot be biamed for. 

Another refinement related to getting informatton in additton to opinions of stakehoklers: 

• Examining whether a factor analysis of measurements from various outcome areas will 
produce factor structures that correspond with the conceptual distinctions in the 
models in order to make a final deciston regarding issues. 



Sample Comments and Cuotatlons 

That the assumptions on which the models rest are so expltoitly pointed out is of especially great 
merit since they ilkjminate the value posittons underlying the models." 

"Making explicit your assumptions is an indicator of the thoughtfulness with whk:h you are going 
about your work and I applaud you for being so clear about them.** 

"I thought highly of the fundamental assumpttons. Very well done, partk:ularly the view of special 
educatton as capacity buikiing." 



Patterns in Responr 9S About Outcomes Assumption^' 



Assumption 1: 



A Model of outcomes Is needed for all students, and at the 
broadest level, should apply to all students regardless of the 
characteristics of Individuals. 



Number Responding to Assumption 1: 19 



Support 



Ten respondents (of the 19 who commented on this assumptton) explk:itly stated that they 
supported Assumptton 1, that the indicators for students with disabilities shoukl be the same as 
for those who do not have disabilities, as being sound and appropriate. Supporters of this 
assumptton thought it was consistent wHh irAegratton kJeotogy, stressing the similarity between 
students with and without disabilities and emphasizing the belief that aH students can learn. 
Several respondents pointed out that at the broadest level, persons with disabilities and persons 
not seen as having disabilities share common goals and outcomes. 
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Conce/ns 

Concerns focused on the inclusiveness of using the term all students and its implications for 
specific populations of students particularly those with severe disabilities such as those who have 
blindness, autism or are permanently unconscious. Concerns in this area included: 

• Stressing similarity makes one blind to the problems and special needs of students 
with disabilities. Indicators must attend to the "speciar outcomes, such as mobility for 
those with bliMness, communication skills for those who have hearing impainments. 

• Examining how realistic it is to relate to the abilities and needs of all populattons, given 
the diversity within the special educatton populatton. 

• Consklering the confusion created by refening to all students, particulariy since low 
SES, disabilities and negative educattonal outcomes are highly associated with 
students in special education. 

• Attending to the tmplicatk>ns this assumptton has for the models. Concern with 
moving to a generic system that has limited relevance to the very audience we are 
trying to reach (i.e., students with disabilities). Some outcomes, for example, may be 
unique to subsets of the populatton. 

Other related concems included: 

• Considering the value of obtaining good outcome measurement on different 
subgroups as a valuable tool in monitoring equity in educatton as well as effectiveness. 

• Linking the model (if indeed it does address a// students) to existing indk^ator systems 
and data bases and other efforts to define and collect data on these variables. 

At least one person suggested that the paper shouM make it explk)it that Assumption 1 reflects 
"special educatton thinking" that is not necessarily wktely hekJ. 



Suggested Alternatives or Refinements 

Two different approaches to dealing with the issue of having the same outcomes for all students 
were represented. One approach was to think of outcomes as the same but with different 
indk:ators (measures) according to student characteristk;s. Another was to think of outcomes 
aiong a continuum of expectations where the goal is to move everyone forward from their starting 
point. Specif k: recommendattons included: 

• Adding an additional assumptton that relates outcomes for all students to societal 
needs for an effective and productive citizenry. 

• Suggesting that an outcome is relevant if its attainment woukJ increase opportunities 
for participating and functtoning in various environments. An outcome still applies 
even if a disability precludes one from ever attaining the outcome or attempting 
performance on an indk^ator of that outcome. 



Sample Comments 

'^Please do not have a different set of outcomes for students with disabilities than for others; this 
woukj set the whole fiekJ back. It may be necessary to woric toward the outcomes in a different 
way for some persons with disabilities, but the basic outcomes are not different." 

"..exclustons (of special education students from requirenrients or demands for higher education 
standards) will consign special education students to the tow skill/tow pay types of jobs. Our 
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benevolence will ultimately hurt the students we are trying to protect! The issue is to set 
appropriately high expectations for each student. Most of the cunrent standards projects...do not 
even mention special education students anywhere in their concern that all students achieve 
higher standards." 

"I share your first assumption that a single nrxxlel of outcomes is possible and prefenred, rather 
than assuming that kids with disabilities and their educational programs need to be assessed in 
some separate way. Different outcomes may have different emphases for different kids. It may 
also be true that we emphasize a parttoular outcome just as nujch for kkjs with disabilities, but that 
the categories or measurement for it are expanded." 

"You couW think of a single continuum of expectatfons, with "adaptive testing" depending where 
on the continuum students are performing. We can define the domains on which we want 
students to advance for all chikJren, but provide depth of measurement at the points in students* 
proximal zones of devetopment." 



Patterns in Testimony on Outcome Indicators 
Assumptions 2, 3, and 4 



Assumption 2: 



Indicators of outcomes for students receiving special 
education services should be related, conceptually and 
statisticaiiyi to those identified for students without 
disabilities. 



Assumption 3: 



Indicators should be unbiased with respect to gender, culture, 
race, and ottier characteristics of the dhrerslty of students in 
today's school population. 



Assumption 4: 



While indicators should meet research standards, those that 
do not could still be used. 



Number Responding with Comments on Outcome Indicators: 18 < 
Support 

General support regarding the assumptions underlying indicators used to measure outcomes was 
expressed in terms of their relatedness to the assumptions underlying the comprehensive 
system of indicators and the broad approach taken to indicators making outcome areas inclusive 
for all students. Very few of the 18 respondents who commented on this set of assumpttons 
provkled statements of support but many more raised concerns regarding the need for greater 
clarifk:atk>n. Of the three assumpttons, assumptton 3 regarding unbiased indk^ators, produced 
more than one supportive comnnent, with three resporKlents stressing its appropriateness. 
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Concerns 

Some general concerns about these assumptions were raised, but ^or the most part, respondents 
commented on specific assumptions. The general comments a.o included first, followed by 
concems raised for each of the three assumptions. General concerns included: 

• Needing to be aware of how the distribution of outcomes would look when schools are 
serving all children well. This produces additional assumptions about what will happen 
to population variance on given measures when this is the case. 

• Defining the territory of NCEO*s project to avoid and getting off target and focusing on 
factors thought to be dispositional of desired outcornes, rather than the outcomes 
themselves. 

• Recognizing there is nonequivalence of information and nonequivalence of outcomes 
when individuals need to attain compensatory skills to reach or denfK>nstrate an 
outcome. 



Regarding Assumption 2: 

A few concems and suggestions were raised about Assumptton 2, whk:h suggests the need for 
conceptual and statistical relationsh^s between indicators of outcomes for students with and 
without disabilities. These inckided: 

• Considering whether Assumptton 2 is really the same or different from Assumptton 1 
(i.e., a model of outcomes applies to all students). lndk:ators shouki be the same if the 
model applies to all students. 

• Viewing this assumptton as Inappropriate if the intent of the assumption is to establish 
nonnative parameters to the indicators. 

• Seeing the impltoations of this assumption as Inconsistent with prevkHJS content in the 
Worthing Paper that views special and regular education as part of the same system. 
This assumptton may be misinterpreted as viewing separate systems as okay. 



Regarding Assumption 3: 

The comments about Assumption 3, the use of indicators that are unbiased and reflect the 
diversity of students in today's schools, ranged from "who can argue with unbiased?** and the 
need to clearly specify what ''unbiased'* means to a concern that given the inequities in our 
educational system, having "unbiased indk^ators" may be overstating our capabilities. Comments 
included: 

• Acknowledging that any differences in outcomes for different subpopulations will raise 
the issue that measures are biased. 

• Recognizing that it is good to make an effort to fairiy represent the accon)plishments of 
a diverse populatton coupled with a need to clearly specify what is meant by 
"unbiased" and a concern as to whether it can be done. 



Regarding Assumption 4: 

Of the comments that were received about the assumpttons underiyiiig the indtoators of 
educattonal outcomes, most were about Assumptton 4, that indtoators dkJ not need to meet 
research standards. This assumptton received the most comments (n»8), of those relating to 
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indicators^ in terms of a need for greater clarity of definition such as how research standards relate 
to validity and reliability. More than one reader suggested that this assumption might raise more 
problems than it solves, by opening to door to unreliable and invalid measures. Comments 
included: 

• Recognizing the possibility of misconstruing the term ''research standards" in such a 
way that the minimum need for reliable and valid measures and procedures and 
outcomes is ignored. 

• Acknowledging the limitations of existing systems such that nruich would need to be 
done to construct a system of indicators that would meet research standards. Few 
existing data systems assure accuracy and consistency, giving little choice but to use 
data with accompanying descriptions of shortcomings. 

• Recognizing th;>t an unclear definition of "research standards," while the intention 
underlying the assumption is valid, will result in both false positives and false 
negatives. Use of indicators that do not meet research standards forces a carefully 
accounting for the whys and why nots. 



Suggested Alternatives or Refinements 

The suggestions focused primarily on the need to more clearly define terms, but provided little in 
terms of actually suggesting how to do so. Terms that needed more clarity, according to 
respondents included "unbiased" and "research standards." 



Sample Comments 

"The paper takes up several important principles concerning outcome indk^ators for special 
educatbn." 

"NCEO shouM inform not only special education but also general educatton indk^ators. It is not 
useful or cost effective to use indk^ators that only pk:k up half of the populatton." 



Patterns in Responses to e Compreheiislve System of Indicators 
Assumptions 5, 6 & 7 



Assumption 5: A comprehensive system of indicators should provide data 
needed to make policy decisions at the state and national 
levels. 



Assumption 6: 



A comprehensive system of Indicators should to the maximum 
extent possible be based on demonstrated functional 
relationships between outcome Indlcttors and Indicators of 
educational inputs, contextual characteristics, and processes; 
however, valued indicators may be included even If functional 
relationships have not been established. 
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Assumption 7: A comprehensive system of Indicators should be flexible, 

dynamic and responsive to review and criticism, changing to 
meet identified needs and future developments in the 
measurement of inputs, contexts, processes, and outcomes. 



Number Respon(£Ing to the Comprehensive System of Indicators: 20 
Support 

Respondents shared several positive comments about these indicators, spread across each of 
the three assumptions that relate to developing a comprehensive system of indicators. 
Comments of support ranged from **nicely stated** to statements of support that had a qualifier 
ryes, but") attached. Those concems are summarized in the followir^ section. 

Concerns 

Concems were addressed primarily to specific assumptions. One suggestion was raised 
concerning making Assumption 5, having indicators that provide data for state and national policy 
decisions, and Assumption 6, relating to using indicators that are/are not based on functional 
relationships between inputs, contexts and processes, simpler and less ambiguous. Specific 
concems are included in the following sections. 

Regarding Assumption 5 

The most frequent suggestion regarding Assumption 5, that a comprehensive system of 
indicators should provide data needed to make policy decisk>ns at the state and nattonal levels, 
was to include the tocal level as well. In fact, of all the suggested refinements and 
recommendations that were suggested by readers in this area, few did not specifically comment 
on the importance of indicators at the k)cal level. Comments that related specifically to 
Assumption 5 included: 

• Recognizing that the usefulness of an indicator would seem to be dictated by the 
relationship that can be established between outcome and interventk>n. 

• Including the local school and classroom levels as well as state and nattonal levels, 
stressing decisions at the school level as nK)st important. 

Regarding Assumption 6 

Some readers emphasized the part of this assumption that acknowledges the importance of 
establishing functional relationships while others commented about the part of this assumption 
that acknowledges the reality that these relattonships often do not exist. Comments focused on 
the need to be vigilant at trying to kJentify retattonships as well as how limiting it woukl be to design 
a system that needed to wait for demonstrated relationships. Readers suggested: 

• Following a careful review and determination that indicators make sense if indicators 
are included for whtoh functtonal relattonships have not t>een estatrfished. 

• Considering whether using valued indicators when no functtonal relationship to 
educatton can be established is inconsistent with NCEO's definition of "outcome.** 
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• Being cautious about too grjat a focus on process-type objectives and inputs if they 
shift the focus away from outcomes. 



Regarding Assumption 7 

While there was support for a llexible, dynamic** comprehensive system of indicators that reflects 
a continuous rather than a sinjgle event process, there was also concern regarding the need to 
balance flexitrility with stability in order to facilitate comparisons over time. For one reader, 
*1lexible** was interpreted as a means by which to emphasize the potential adaptability of the 
system to involve local input. Comments included: 

• Acknowledging the tension of trying to find a balance between flexibility and stability. 

• Recognizing that ample stability is essential for cross-time comparisons to measure 
improvement; changing the data collected each year Is not useful. 

• Being aware of the need to continue to release information as things evolve and 
change. 



Suggested Aiternatives or Refinements 

Several respondents raised the issue of including local indicators as well as state and national 
indicators. The concerns about that issue as well as about other assumptions regarding a 
comprehensive system of indicators resulted in the following suggestions: 

• Making an effort to suggest how SEAs can fadlrtate LEA efforts and vk:e versa in order 
to inform policy devetopment at the kx^al level and to make the comprehensive system 
of indk:ators functional and useful on a local level. 

• Engaging local educators who are involved in the data collection this system will 
require by provkling meaningful feedback on data collected and reported at the k>cal 
as well as at state and national levels. 

• Adding to Assumption 7 a statement that recognizes a need to make a commitment to 
stick with some data elements that remain the same since data collection that changes 
year after year is not useful 



Sample Comments 

Regarding Assumption 5: **AII outcome domains in (the) Model have applk^ation at the local district 
and classroom level. I do not note an indtoator that district/buikling level poltoy/deciston nrnkers 
would not find interesting, relevant, and helpful in evaluating and improving educational 
experiences for chiMren, youth and adults with disabilities.** 

Regarding Assumptton 6: "I tend to bend over backward to be cautkxjs about process-type 
objectives and inputs, because I feel they have led us away from facing outcomes. For too tong 
we have been counting noses, e.g., are they in a resource room and regular class part-time?, 
rather than outcomes-what dkl they learn? How do they feel about themselves. We have 
soothed ourselves with data that suggested something was going on, rather than tooking at the 
tonger term results.** 

Regarding Assumptton 6: **l am glad to see variables included even if they doni con'etate with 
outcomes so it more fully describes the educattonai context and is less likely to miss secondary 
relationships. The correlations can be very misleading. Things diffkxjit to measure, such as 
parental interest and involvement are often omitted from indtoator systems." 
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Regarding Assumption 7: **Changing what gets measured from year to year means that 
improvements over time can not be documented. t*m rK>t sure how you balance 
dynamism/flexibility with stability/comparability, but acknowledge the tension between iriem.** 



Testimony on Outcomes Model: Model 1 vs. Model 2 
Number Responding with Testimony on Models: 40 



Support 

Several respondents suggested both nxKlels were appropriate and did not state a preference for 
one model over the other, while two respondents suggested a third alternative. Of the 17 who 
expressed a preference, approximately twice as many preferred Model 2 (n»11) over Model 1 
(n=6). The following comments and concerns are separated by responses to each of those two 
Models that are presented in figures 1 and 2. 



General Comments on Models 1 or 2 

There were many reactions to the two Conceptual Models of Outcomes and Indicators that were 
proposed in the Working Paper. In addition to the preferences for either of the two nrxxJels or for 
entirely different conceptualizations altogether, some readers suggested a need for clarification 
of components within the models. Some asked for clarificatton of components of particular 
models, such as the difference between presence and participation, between accomplishment 
and contribution, others suggested that greater emphasis be given to ceilain comporierrts (e.g., 
knowledge, skills, attitudes) that are direct outcomes, and that indirect outcomes such as 
participation, presence, contribution and satisfaction grow out of these. 

• Not sure of the differences between the two, txit liked the broad outline of input- 
process-output model as the basis. 

• Both models include all of the outcomes that shoukJ be included. 



Support 
Model 1 

• It is easier to deal with the outcome areas mentioned separately in model 1 than it is to 
view them as multiple dimensions of accomplishments as in nxxiel 2. 

• Model 1 is presented in clear, commonly understood language that lends itself to 
implementation and monitoring. 

• Model 1 makes sense and has an economy and elegance that is powerful. 
Model 2 

• Model 2 is simpler and easier to understand and operattonalize. 

• Model 2 is useful in placing the 6 national goals within the larger circle of aspirations we 
have for the educational system. 
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• Model 2 presents the most understandable model to the field and is in higher 
congruence wKh the America 2000 initiative. The outcomes domains are simple and 
direct Ixit comprehensive for the purposes stated. 

• Model 2 appears to have an increased emphasis on what the public considers as more 
important outcomes. 

« Prefer the intent of Model 2 in that outcomes are viewed as acconrtplishments and that 
presence and participation are included. 

• The broad strokes (presence, participation, cognKive, attKude) are generally okay. 

• Model 2 relates better to the issue of avoiding general education policy with only 
limited recognition of special educatton students. 



Concerns 

General (Model l or 2) 

The differences between the models were not clear to some respondents. One issue seemed to 
be the possible perception that Model 1 suggests equal value for all outcomes. This issue was 
addressed from both points of view, with one respondent suggesting that Model 1 was less 
desirable because of the suggestton of equal value for all outcomes. CXhers suggested the 
importance to tx)ards and state policy makers of considering them as equally important and 
acknowledged that although some may perceive the imptoatton of "equal inrtportance** in the 
model by listing separate outcome areas K did not necessarily carry that connotatton. 

• Concem was raised as to whether the consideration in Model 1 of accomplishments as 
one outcome as contrasted wKh Model 2's approach of consklering outcomes in nine 
categories to be sub-categories of accomplishments is a semantic/organizattonai 
difference or one that may make a real difference. All Kems under accomplishments 
may confuse the other crittoal educational outcomes wKh student accomplishments 

• Neither addresses student characteristk)S and family make-up, both of whk;h greatly 
impact outcomes. There is a need to recognize the influence of support for teaming in 
the family environment and to cover unic^Je skills that are a focus for students with 
disabilKies. 

• Need to address social issues that intertwine wKh and have an impact on educattonal 
outcomes. 

• Both models are missing the links to inputs; it is a circular policy process. 

• NeKher makes a specifk: reference to motivation for life k>ng teaming as an educattonal 
goal. 

Other general concerns regarding the models related to nrx>re clearly defining the differences in 
terminotogy including the difference between presence and parttoipation, and accomplishment 
and contributton. Conskleration of these may need to be made in terms of a continuum, not a 
statk; dimenston. Some respondents supported incluston of presence and partk)ipatk)n as being 
associated with student outcx>mes while others acknowledged their role but questtoned their 
association with outcomes. 
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Several respondents commented on components included in the Model of Education that was 
presented as Figure 1 in the working paper, including one who thought it wasXhe (NCEO) model. 

General suggestions included: 

• Moving staff development under organizational resources or under personnel, not 
school programs. 

• Focusing on resources and processes within and across school, community, and 
home settings. 

• Recognizing the usefulness in differentiating between curriculum (what is planned and 
available) and instruction (who actually takes courses and what is actually taught). 

• Adding information on the relationships/intenrelationships among elements in the 
model. Outcomes can relate to each other in Important ways at any single point. 

• Considering the influence that child characteristtos have on what the school does with 
a student that go beyond influencing processes to influencing outcomes. 



Model 1 

• Meaningfulness in participation must be considered rather than the number of 
community events. 

• Need to clarify that presence inckides "access" and use engagement ot the social and 
academk) environment as an indicator of partk^ipatton. 

• Regarding participation, the model needs to address the extent to which all students 
participate, not just special educatton students. Model 1 raises concerns about the 
focus on academto areas only. 

• Mixing outcomes with processes is a concern. 

• Implies a parallelism between outcome domains that does not exist, that is, implies 
equal value of presenced participation with accomplishments for example. 



Model 2 

• Listing "contribution" as a form of accomplishment (as a sul)ordinate, not coordinate 
relattonship) is confusing. 

• Stretches the concept of accoiDplishment beyond the point of usefulness to make it 
accommodate such domains as attitudes, presence etc. 

• Lacks hierarchy and has too many elements, making it more compltoated and dffficult to 
specify. 

• Model 2 is too generic to be helpful to special educators and too simplistic to 
communicate to regular educators. 
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Suggested Alternatives or Refinements. 



Several respondents suggested refinements of a Xonceptual Modei of Outconrie Areas, 
Donnains, Subdomains and Indicators** that was presented as part of Model 1 in worki^ig paper 1 
(see figure 3). The concerns centered primarily around the need for subdomains under the areas 
that are included, or the need for combining subdomains such as writing, literacy and language 
into "communioation.** Another respondent suggested eliminating the figure since it doesnl add 
to the working paper and oversimplifies the complex interactions among outcome domains and 
outcome areas. 




Figure 3. Conceptual Model of Outcome Areas, Domains, Subdomains, and Indicators 



A number of the suggestions were linked to post school outcomes, with respondents suggesting 
that it may be important to make a distinction between outcomes for which the educational system 
can and cannot be held accountable and that it may be beneficial to start with post school 
expectations and wori< back to improve schooling. Still others suggested specific outcome areas 
that should be included in the model. 

Specific suggestions included: 

• Focusing more heavily on kr)owledge, skills and attitudes and treat other things as 
functionally related to these direct outcomes. Other things are enhanced by the 
knowledge, skills and attitudes that are the focus of the model 

• Examining the inputs, processes and outcomes as well as contextual variables and 
addressing the bi-directional nature of the relationships among various elements of the 
model. The process of leaning may be as important as the outcome for some 
students with special needs. 

• Reexamining assumptions and procedures in offering students with learning 
disabilities an education in light of the high drop out and non-completion rates and 
poor post-secofKlary involvement. 
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• Giving more emphasis to measurements of self-concept, social interactions, role- 
modeling, feelings of isolation. Opportunities for success are perceived as critical to 
higher rates of post-secondary education and employment for students with severe 
physical disabilities. 

• Recognizing that some things important for students with disabilities might not be 
important for those without disabilities (presence, participation, unique disability 
specific outcomes). Considering the unique access and equity/inclusion issues. 

• Adding motivation for life long learning rather than having to infer it from presence and 
participation. The model should reflect high personal expectations for growth and 
development. 

• Considering three general outcomes critical with respect to schooling that are essential 
for all but can be achieved along a continuum from minimally to very high levels. These 
include the skills, knowledge and attributes for (1) language literacy, (2) constructive 
social behavior and (3) social dependence. 

• Using an alternative model that includes adding critical alternative outcomes for 
chiMren and youth with disabilities such as mobility skills, signing and use of adaptive 
technology. 



Sample Comments 

"Add a 'how to' emphasis that shows people how to examine their own tocal situatk>ns, primarily at 
a school level." 

"Look first at desired and valued post-school and system outcomes to estabEsh the f rameworic for 
culminating outcomes." 

"Start all outcon)es with post-school expectations and woric back to improve schooling." 

"Consider that the context and ideas about resources will be changing (school effectiveness, 
instmctional effectiveness, families, communities, potk^ies). It is in this context that we should lay 
out the conditton of schools, of home, etc. that seem important and design rating systems for the 
several components and their interacttons.** 

"Right or wrong, general education looks almost exclusively at accomplishments. A model, with 
categories such as presence and parttoipation, needs to took tess special in order to be palatable 
to general educators where such goals are often assumed but not reiterated.** 



Proposed Model 3 

Based on preliminary responses to Models 1 and 2 a third alternative was presented as Model 3 
(see figure 4) to NCEO's advisory committee in October, 1991. Model 3 (which was not included 
in NCEO's Woriung Paper 1 ) and very Bmited feedback regarding this nxxiel is presented here. 



Support 

• Model 3 is better from a devek>pmental perspective. 

• "Enabling outcomes" are perfect for measuring the contributtons of special education. 
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General Testimony on Issues 

Respondents were asked to address seven issues. A few individuals provided general 
comments on the issues. These comments supported the need to consider all of the issues but 
did not assume that all needed to be resolved prior to addressing indicators. One suggestion was 
made to "lead off** with these issues and focus efforts at developing a process that could be used 
by schools, LEAs and SEAs attempting to identify process and outcome quality indicators. 

Issue 1: Intended vs. Unintended Outcomes 
Number Responding to this Issue: 28 

Support 

Of those who commented on the issue of intended versus unintended outcomes, the responses 
were split as to whether to focus on intended outcomes or on both interKled and unintended 
outcomes. Of the 10 respondents who favored a focus on intended outcomes, more than half of 
them emphasized a major or primary focus on intended outcomes, but added that unintended 
outcomes are important and may warrant a secondary or later focus. Half of those who agreed that 
unintended outcomes were irrportant commented that both were critical, while the other half 
strongly emphasized that unintended outcomes were critical. Responses that favored each of 
these positions are included below: 

Intended Outcomes 

• The major focus (first priority) should be on the intended outcomes. 

• Target intended outcomes. Be alert to unintended outcomes, but don*t expend major 
energy developing systems that define them or collect data about them. 

• Inclination is to favor the collection of information on the inter)ded outcomes as a 
primary objective, while pursuing the collection of information on unintended 
outconf)es on a wore limited basis. 
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• Intended outcomes should be regularly monitored with data periodically collected to 
identify unintended outcomes. 



Unintended Outcomes 

• Strong belief that unintended outcomes should be addressed. 

• Unintended outcomes, such as the enx>tional impact of increasing standards, must be 
a part of the information collecting process. 

• Unintended outcomes are essential to evaluate, making the issue of subdomains 
particularly important. 



Intended and Unintended Outcomes 

• Like the notion of both intended and unintended outcomes. 

• Initial reactton is that informatton on both need to be collected. Otherwise the risk is of 
only looking for what we hope/intend to find and miss signifk)ant data. 

• Public pressure seems to vok:e a desire to invest in interventtons and then know the 
results. Concentration only on intended outconnes woukl appear to be manipulative or 
dishonest. 



Concerns 

The concerns that were raised included comments on recognizing that unintended outcomes can 
be negative as well as positive and the difficulty in klentifying unintended outcomes and in linking 
unintended outcomes to the educattonal process. None of the concerns were about intended 
outcomes. One respondent suggested it is far too early in the process to make a decisbn on this 
issue. 

• Unintended outcomes can be either positive or negative but need to be recognized. 
A process to improve reading may improve attendance or self esteem, for example. 
Finding things you are not k)oking for (i.e., unintended) partk^ularly if the effects are 
negative wouM be diffk^ult. 

• Unintended outcomes, such as the emofional impact of increasing standards, must be 
a part of the informatton collecting process. Cautton if increasing standards are at the 
expense of emottonal well-being. 

• There are so many unintended outcomes of the educattonal processes that klentifying 
them and drawing cause-and-effect conckistons wouU be a very diffkxjlt task. 

• It is hard to link intended outcomes to the input, process and context components. 
How wouU unintended outcomes be linked to these? 

• There is some difficulty with the definition of intended and unintended. 

• Unintended outcomes wouM be diffk:ult to klentify and care must be taken not to 
assign causality. 

• Unintended outcomes wouU serve little purpose if comparison of data on outcome 
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Suggested Alternatives or Refinements 

• Once various unintended outcomes are conceptualized and defined, indicators to be 
included might be decided from an estimation of social importance of the outcome and 
the functional relationship the outcomes show to school variables. 

• Unintended outcomes that are consequently identified can be listed as "cautions** and 
be Incorporated Into the model at a later date or serve to illuminate the need for open- 
ended data gathering instmmentation. 

• While difficult to include unintended outcomes a priori it would be nice to have a model 
and data collection procedures that are sensitive to unintended outcomes that occur. 

• We need to measure in both directions. Positive vs. negatr;e effects is related to the 
issue of intended vs. unintended. The values business drives much of current 
practice. 

• If an outcome is suspected of t>eing negative, a hypothesis should be developed to 
determine whether indeed there is a causal relationship. 



Sample Comments 

**Maintaining some degree of focus on unintended outcomes may lead to a xx)ore accurate 
reflection on analyses stemming from the intended outcomes." 

"Initial reaction is that infomnation on both need to be collected. Otherwise the risk is of only 
looking for what we hope/intend to find and miss signifk:ant data that might indicate either the 
success of a given approach or the need to change direction." 

*There are so many external influences on effects of schooling. Schools cannot be blamed for 
all.** 



Issue 2: Direct vs. Indirect Outcomes 
Number Responding to this Issue: 28 



Support 

Over half of those who commented on this issue were supportive of collecting indirect outcom^ 
data. Only four of those who addressed this issue stressed that only direct outcomes shoukf be 
considered. Several specified the importance of indirect outcorres for students with disabilities, 
in partteular and listed several indirect outcomes such as unemployment, independent livirig and 
post secondary educational opportunities. A few suggested that the focus is a matter of priorities 
and that regularly monitoring direct outcomes while periodically collecting data to identify indirect 
outcomes reflects an important emphasis. 



Direct Outcomes 

• Direct outcomes should be the major focus. 

• The primary focus should be on direct outcomes. Direct outcomes, however, should 
be related to indirect outcomes. 
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Indirect Outcomes 

• Information about indirect outcomes is needed to facilitate our understanding of the 
effects of integration on academic and social learning. 

• indirect outcomes may in the long run t>e the most important. Having longer-term 
outcomes, while difficult to ascribe to programs that occurred earlier, are very 
important. 

• Skill and concept building will often lead to transfer and other developmental 
opportunities that were unforeseen. Negative aspects need to be monitored as well. 



Direct and Indirect Outcomes 

• Need to address both direct and indirect outcomes, tracing both interim outcomes and 
distal outcomes. The core effort must focus on the culminating outcomes for all 
students. 

• Particularly for students with disabilities, it is important to consider both direct and 
indirect outcomes. 



Concerns 

Several raised concerns about their disagreement with the distinction made between direct and 
indirect outcomes presented in the working paper. One respondent suggested that the best that 
can be hoped for is to account for immediate post-school outcomes including links to adult sen^ice 
systems. 

• Disagree with the distinction made between direct and indirect. Schools shouM be 
responsible for post school events (these are not indirect) but are shared 
responsibilities of the school with families and society. 

• Whether an outcome is direct or indirect may be a false dichotomy, depending on the 
goals of education. 

• The tenninology may hinder discussion of the issue. An indirect outcome may be the 
synthesis and application of the direct outcomes. 

- Employment as an example of "irKlirect* is unfortunate. Happiness or job satisfaction 
might be better examples. 

• When direct outcomes can be measured, spending scarce resources on indirect 
outcomes may not be warranted. 



Suggested Alternatives or Refinements 

• Indirect outcomes that shouM be included shoukl be based on an estimation of their 
social importance and the functional relationship to other school variables. 

• Include outcomes that address whether students are prepared to enter the workforce 
or for adult life. 

• Change the label of Indirect" to "impact to avoid some unfortunate connotations. 
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• Perhaps the issue is about the relative location of the outcome on a continuum ranging 
from rote replication of a skill taught to resolution of new problem. This issue parallels 
the current discussions on "authentic testing**. 



Sample Comments 

**The goai of education is to prepare kids for life. Many of those values are indirect outcomes, but 
we need to recognize them and the impact they have upon life.** 

"I think schools should be responsible for post school events (employment, attending 
postsecondary educatton, experiencing a reasonable quality of life). Agree these responsibilities 
are shared with others (student, family, society) but this does not release the schools from 
responsibility nor should these outcomes be considered 'indirect.' 

'^Because there is so much controversy now on what the goals of educatton are, we must took at 
both direct and indirect goals.** 

**lt is the k)ng-term consequences of the direct outcomes which will tell us if the special educatton 
system is successful or not. If this is an accountability system, it may be important to avod this; 
however, since I assume that this is not the purpose, then such links between direct and indirect 
outcomes, even if tenuous at best, can be important.** 

"I see post-school happenings as very relevant and 1 fear that viewing them as indirect outcomes 
may lead educators away from the real world.** 

**! disagree with, but recognize the national climate that considers post-school outcomes such as 
employment to reflect the results of schooling. The times are such that we have no choice but to 
report post school outcomes.** 



issue 3: Same vs. Different Indicators 
Number Responding to this issue: 26 



Support 

Almost half of those responding to this issue stated the need for the same indicators, with some 
of them expressing a qualification that if the same indicators are used, a different emphasis may be 
necessary for some students with disabilities. Another 20 percent stressed the need for having 
both same and different indicators, with the result that close to seventy percent of those 
responding to this issue favored including indicators that were the same for those with and 
without disabilities. 



Same Indicators 

• If we intend students in special education to be part of the same workj as regular 
students, indtoators woukj be the same. 

• Different indicators leave out students with special needs. 



Different Indicators 

• General outcomes must be the same, but with different indicators. 
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• You will have some different indicators for special education, if your model 
encompasses conditions or predispositlonal factors. 

• Progress indicator:, are different t)ecause students doni start at the same place or with 
the same condition. 



Same and Different Indicators 

• To the greatest extent possible, we need the same indicators, but for those with {he 
most severe disabilities there may need to be 'screening' questions that shift the 
emphasis. 

• Having the same indicators is reasonable and a good starting point. Both similai and 
different indicators will be necessary to assess the perfomnance of certain groups of 
students with disabilities. 

• There is a core of culminating and post-school outcomes for which all of education 
must be accountable, but at the same time, special education students are different by 
definition. 

• Same and different. CXjtcome measures can be expanded if the specific experience s 
for students with disabilities indicate a relevance. 



Concerns 

Concerns reflected both positions regarding the possible impact of having the same indicators 
and the impact of having different indicators for students with disabilities. With the ramifications 
and strategies for developing outcomes that are either a replacement for, an altemative to, or a 
supplement to general education standards being different, one respondent suggested it was 
essential to be clear about the purpose from the outset. 

• Separate indicators will continue to diffuse the issues. The same indicators are okay 
even though special education students will do less well. 

• Different indicators will sefve to promote spe< -al education as a separate and unequal 
educational system. 

• A separate assessment system only works to reinforce the separateness arid 
differences persons with disabilities, professtonals and families are trying to diminish. 

• The practk^e of using the same indk:ators for those with severe disabilities as for those 
in regular educatton seems very questtonable. The current assessment practk^s in 
regular educatton that are inadequate to measure student progress of those with 
special needs has a negative emotional impact and is often unfair. 

• The same categories can be used for all if measurement is expanded to inchjde the 
amount and type of cost and supports needed by people with disabilities. 

• Pr^obably need a bit of both. Students with disabilities are different and they leam 
differently but the concem is their need to live in the sanrte society. 



ERLC 



25 

30 



NCEO Synthesis Report 



Suggested Alternatives or Refinements 

Several of the suggestions focused on how to accommodate the unique needs and 
characteristics of students with severe disabilities, primarily by adapting the system of indicators 
even if they are inttially held accountable for the same indicators as students in regular education. 

• Keep genera! outcomes the same, but use different indicators that allow for different 
ways for people to demonstrate the accomplishments. 

• Keep indicators the same, but use unique assessment systems and data requirements 
if necessary. 

• Keep indicators the same for those with mild disabilities but recognize that different 
kinds of data need to be gathered on students with severe disabilities. 

« Expand the definitions, but keep the same indicators. Emptoyment outcomes could 
include supported and day activity emptoyment as well as competitive employment, for 
example. 

• Adopt a principle of adaptive measurement in order to ask different questtons of 
different individuals, but at the end have a comparable score. Attempt to measure the 
same knowledge, skill and attitude areas for ail students, but use different questtons or 
items to reflect a student's status in regular or special educatton. 

• Shift from the notk>n of same vs. different to conceptualizing overiapping informatton 
or a central band of informatton to be collected with additional informatk>n to be 
collected for students at the extremes (with severe disabilities or gifted and talented). 

• If a student will not pass any of the same indtoators, go to a different set of in-depth 
questtons, partk:ularty for those with severe disabiltties. 



Sample Comments 

"I believe the concept of different indicators has the unfortunate effect menttoned as a rationale 
for establishing your Center. The reform movement has left students with special needs out 
because they believe the indicators shoukJ be different." 

"Using the same indk^tors will encourage us to maintain appropri^e expectattons for all of our 
students. However, the emphasis or approach may vary." 

*'l strongly support the partnership between special and general educatton. However, I feel that 
although general educatton has presented some outconf)es for all students (economtoally and 
polittoally driven), we still require a special set of indtoators for the severely and multiply 
handtoapped. The set of indk^ators will compliment the general set." 

"We have to know the extent to whk;h our kkls are achieving the outconf)es that soc^ty values. 
Most importantly, we have to show where our kMs stand in relatton to the nattonal goals. At the 
same time, our kkJs are different by definitton. We have to know if we are accomplishing the 
interim outcomes that 'enable each indrvkiual with a disability to live as nomial a life as possble'." 

"The template metaphor doesn't work, speciftoally the attempt to overlay outcomes on an 
underlying system that is skewed in favor of 'normar or 'gifted' chiklren. The existing system is 
out of alignment. We all have a clear misston grounded in the belief that all chiklren have the right 
to be educated with dignity and high expectattons in a convnon supportive environment." 
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**Elements of instruction and curriculum are modified/adapted to meet the needs of students with 
disabilities. Would it violate this philosophy to modify an assessment system? Some indicators 
will simply he not applicable for certain students." 

*To the extent possible, even students with severe disabilities should be afforded the same 
opportunities and therefore the same outcomes as peers in regular education. It can only 
strengthen the system to hav^ it involved with the tremendous accountability movement in 
regular education. The outcomes specified in the national goals should clearly be reflected in 
the outcomes selected by NCEO." 



Issue 4: Category Specific vs. Noncategorical Indicators 
Number Rosponding to this issue: 26 

Support 

Nineteen individuals responded to this issue with comments of support for either categorical or 
noncategorical indicators. Virtually all of them supported using a noncategorical approach, with 
three fourths of them strongly supporting only a noncategorical approach, while the others 
suggested the need to have both types of indicators. One suggested that it was not a 
"categoricar issue, perse, but a question of scale and detail. 

• To the maximum extent possible, we must stay noncategorical and keep indicators at 
the broadest level that applies to all students. 

• The use of categories only reinforces separateness and differences; data should not 
be differentiated across disability. 

• As nfHJCh as possible, use noncategorical. Keep categorical components (outcomes 
and indicators) minimal. They should be defined in the same way that schools are 
defining goals for all kids. 

• Noncategorical indicators are all right as tong as they are inclusive and contain alt 
possible characteristics. Indk^tors that are constructed as category-specific could be 
facilitative. 

• Adoption of noncategorical indicators does not devalue the importance of category- 
specific indk^ators. 

Concerns 

Concerns focused primarily on the impact of categork^al approaches on those with multiple 
disabilities and the usefulness of limiting the focus to data that only apply to all chiMren. 

• Collecting categorical data encourages differences and separateness. Within special 
educatton, categorical approaches leave out students with various (multiple) special 
needs. 

• There is a dilemma in that if data are limited to those that apply to all categories of 
chiklren, we have a generic sfil of data that jdb3fi& not recognize the presenting 
characteristk^s of chiMren an youth. We do need data that are useful and help us know 
what to do after we collect them. 
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• There are some differences by category {particularly for studerrts who are blind or deaf 
or have severe-profound disabilities), but otherwise it is only timing and intensity or 
density differences that exist, not essentially different kinds of instruction or 
environments. 

• Without differentiation by category or level of functioning there is risk of overtooking 
important educational results achieved through special educatton. Valid decisions of 
how differentiated these indicator subdomains need to be requires some empirical 
test. 

Suggested Alternatives or Refinements 

• The inclusion of category specific indicators may lead to the accumulation of 
information with implications for the development of policies designed to meet the 
educational needs of specific populations. The stance taken will reflect the depth to 
which NCEO is willing and able to probe the educattonal performance of all students. 

• Use noncategorical indicators, but altow data to be summarized by disability categories 
in order to discover whether the outcomes are the same across excepttonalities. 

• Use category specific indicators for students with nxxlerate to severe disabilities only. 

• Think in terms o\ a central band of overiapping infonnatton, in some respects it wouW 
be the same, in others different. 

Sample Comments and Quotations 

"All students shoukj be assessed on a common set of outcome indicators." 

"Any distinction across categories or across severity will limit the possibilities for irxJivkiual 
students. The differences are on an indivkJual student basis and not on a categorical basis. 
Some of the outcomes could be achieved by some students with disabilities incWentally, and by 
other students only with intense effort. Strive to woric from noncategorical indfcators." 

"The use of category speciffc indk^ators seems contrary to the fundamental positton stated in this 
paper that "special educatton is a capacity buiWing resource for general educatton." 

"Categorical indfcators leave me asking 'So what?* and wondering if we are programming certain 
subgroups to be the 'best of the worst* ." 



Issue 5: indicators Differentiated by Severity of Disabiiity 
Number Responding to this Issue: 27 

Support 

Of those who commented with statements of support regarding this issue, it was equally divided 
between those who supported the need for indtoators differentiated by level of disability and 
those who thought the indtoators needed to be comprehensive enough to address the needs of 
all students. Several who wanted to Ignore severity of disability as a factor, however, cormiented 
that in the area of mental ability, that is for those with severe mental retardatton, differert outcome 
indicators might be required. 
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Differentiated by Severity of Disability 

• Severity of disability definitely impacts student progress and in order to truly represent 
the outcome it should be addressed. 

• Indicators would be adjusted to severity of disability since all indicators should begin 
with the individual as stated in the lEP. 

• Different kinds of data need to be gathered on students with severe disabilities. The 
data must t>e valid and reliable for each degree of severity. 

Not Differentiated by Severity of Disability 

• Indicators nHJSt be comprehensive enough for all students. 

• Keep the same. We want generic outcomes aaoss student populations. 

• Discourage the use of severity levels; it is okay if a snfmll percentage of the populatton 
is considered on indicators that are simply not applicable. 



Both 

• Collect the same kind of data for all students while looking at different 
accomplishments. 

• A broad conceptualization that has a set of outcomes for all deaf and hard of hearir)g 
students that is similar to those established for students without disabilities is 
important, however, it might be applicable to have differentiatton for severe mental 
retardation. 

• Ignore all references to severity with the exception of niental ability. There are 
different expected outcomes for those who have profound mental retardation than for 
those who have mikJ mental retardation. 



Concerns 



What we need to be able to say is that student characteristics will mediate or moderate 
outcomes and recognize that some students will never achieve certain outcomes. 

Different kinds of data need to be gathered on students with severe disabilities. The 
degree to which the assessntent system will be different must be contingent on the 
severity of the disability. 

Outcomes for students with very severe disabilities fail mostly into the *'criticai interim 
outcome** category rather than in culminating outcomes. 

A fundamental problem is being overiooked, that is, how severity of disability interacts 
with conclusions about outcomes. 



Suggested Alternatives or Refinements 

There were few specify suggesttons that were stated to address this issue, although in the 
comments regarding support and concerns, it was clear that refinements are needed with 
respect to how the outcomes for students with severe disabilities wilt be addressed. 
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• Use a strength model to stress rate or progress toward growth and development; 
indicators should encompass seventy rather than differentiate by seventy. 

Sample Comments and Quotations. 

**AII Students should be assessed on a common set of outcome indicators.** 

*'Any distinction across categories or across severity will limit the possibilities for individual 
students.** 

**Think in terms of a central t>and or overlapping information, some would be the same and some 
would be different for students with differing abilities.** 

"Indicators may encompass severity, rather than differentiate by severity. The proportion of 
children in the 'projected' range rather than the 'expected' range may be the focus.** 

**lf you only measure outcome status, how will you know when defk)ient performance is due to 
the severity of the initial disability and when it is due to ineffective educattonai programs; likewise, 
how will you tell whether excellent outcomes are due to effective programs or to the 
misidentifk:ation of nonhandicapped children as handtoapped?** 



Issue 6: Indicators Differentiated by Developmental Level 
Number Responding to this Issue: 22 

Support 

The individuals who responded to this issue primarily stressed that devetopmental level needs to 
be addressed and that doing so does not present a problem. Only two indivkiuals indk^ated that 
the indicators (at least for those with mikl disabilities) shoukJ be the same since it cs a matter of 
expectattons or standard setting. 

• Some are consistent across levels, but there is no questton that different kinds of data 
are collected at different levels and that the outcome expectations vary significantly for 
primary vs. secondary level students. 

• Collect different kinds of data by developmental level, with vastly different indicators. 

• Make allowances for devek>pmental stages. 

Outcomes in emottonal, social and cognitive domains are relevant at all levels, only 
indicators may differ. Age or devetopmental outcomes that are specifk: don't present a 
problem. 



Concerns 



ERLC 



Developmental data for special education students would be chronologically 
inappropriate and of little value in the fiekl. 

Different kinds of data will need to be collected on students with disabilities. 

Avoid the potential trap of linkirvg severity and developmental levels, thus keeping 
individuals with disabilities at age inappropriate levels. 
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• Sonie strands can be followed from preschool through secondary school (presence, 
participation) but some cannot. 

• The only way to address this issue is that it depends on the outcomes that are 
established; indicators may or may not vary across developmental level depending on 
the outcomes addressed. 



Suggested Alternatives or Refinements 

• Develop indicators consistent with the expectation of the next environment. It may not 
parallel the notion of indicators differentiated by developmental level, but it seems 
nfx>re fruitful to do so. 

• Keep indicators separate by ages (those strands that do follow from preschool to 
secondary levels) and you'll provide more useful information. 

• If different data are gathered at different points (e.g., for infants, at school entrance), 
then differentiate chronologically, not developmentally. 

• Rate of progress toward growth is preferable to "normal" growth as an outcome. 

• Making a decision In this area depends on the nature of the data collected and the 
reason for its importance. Consider whether there is a point In a student's 
development at which information about particular areas becomes insignificant. 



Sample Comments and Quotations 

"Different indicators will be needed for the same general accomplishments (if woik/employment 
coukJ not be an accomplishment a more general indicator such as (meaningful day activity) would 
be appropriate)." 

"The importance of process as an outcome is particularly relevant here. A strength model of 
outcomes would stress rate of or progress toward growth and devek)pnr)ent as an individual, 
rather than *normar growth and development as an outcome.** 



Issue 7: System Level vs. Individual Level Indicators 
Number Responding to this issue: 31 



Support 

The majority of those responding to this issue (75%) favored collecting both individual and 
system level indicators. Three individuals stressed concentrating on individual level indicators 
while two aclcnowledged the importance of documenting system level effects. For the nK)st part, 
however, respondents simply indicated both were needed or stressed that system level 
indicators grow out of an aggregation of individual level outcomes. 

• These are not either/or issues. Individual outcomes are of interest, but data on 
numbers of individuals provides systems with the opportunity for change and 
improvement. 

• Clearly both are needed. Most system-level indicators are aggregations of individual- 
level ones (drop out rate and percent passing). 
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• There is an ordering of quality assurance levels that begins with the individual and 
advances to a systems level 

• Outcomes serve different functions. The national agenda needs system indicators; 
special educators are interested in individual indicators to address the quality of the 
educational experience for the individual. The issue relates to why outcomes are 
assessed. 

• Both are needed. If you have to favor one, go with the group or system level data as 
they are where policy decisions reside. 



Concerns 

• It is doubtful that any system can make individual and system level decisions 
adequately. 

• All systems level indicators must be individually relevant. 

• A focus on individual level outcomes sustains the traditional focus on individual 
deficits. 



• Issues relate to the level of reporting as well as to the levels at which to collect 
information. 

• Consider what jurisdictions the indicators will be used to nwnltor, such as districts, 
states, full-time placement facilities. Those that serve the broadest populations (i.e., 
those with more mild disabilKies) will score best on indicators. 

• Not all people with disabilities within a system will have indicators of progress from the 
same input. 



Suggested Alternatives or Refinements 

• Focus on individual outcomes that can be summarized by systems, when desired, 
considering system and individual levels separately. 

• Rethink the concept of "system lever indfcators to go beyond exaniples of "kid stuff" 
to questton whether outcomes shouW be defined for things such as consumer 
satisfactton, cost savings and the impact on interagency access. This raises the 
questton of whether we conskler the overall effects the educattonal process and 
student outcomes have on other parts of the system/society. 

• Include indicators at all levels from broad based to specific: natfonal, state, system, 
school, program, classroom, indivkJual. The data will vary in scope, detail and content 
dependent on the validity, reliabirity and usefulness." 



Sample Comments 

"A focus on system level indicators forces us to reinforce strengths, rather than remediate 
deficits." 

"Go with indivkJual level outconries. The issue goes back to why outcomes are assessed. If it is to 
address the quality of the educattonal experience of the individual with a disability then you need 
to assess indivkJual level outcomes." 
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''Ciearty both are needed. Most system level indicators are aggregations of individual level ones 
(drop out rate and percent passing). Taity system level indicators (school dimate, race relations, 
vandalism) are not critical final goals of education.*' 



General Comments and Concems 

General Testimony 

Number Responding with General Testimony: 34 



Patterns In General Comments to the Working Paper 

Of the approximately 47 general concems that were raised, over half were editorial suggestions to 
clarify points or to add more detail about specific areas. Such editorial suggestions included 
adding a focus that places the paper in its historical context, empha^ing the focus of NCEO eariy 
in the document, defining and describing what we want schools to do or produce, acknowle<)ging 
the cost of the effort, simplifying the nrKxtel and clarifying definitions, such as the definition of 
indicators and of outcomes. 

The responses contained a great number of statements of positive support for the efforts and the 
current status of NCEO's efforts to date. The comments reflected both an appreciation of the 
thought-fullness and logic reflected in the working paper and of the comprehensive nature and 
importance of the task that it represents. 

Several of the respondents (about one third of those who res|3onded to the wortcing paper) raised 
issues related to indtoators. One theme within this broad area related to examining the indk^tors 
with respect to how they relate and can be linked to school experience. One consideration was to 
enhance the simplicity of the model by limiting the focus to things that are denx)nstrably related to 
school experience. Other respondents expressed concerns about confuston between indicators 
and outcomes, with some suggesting that the focus shouki t>e on the latter. Specific issues that 
related to indicators included: 

• Concern that the identificatton of *1he list of indtoators" will force people to line up on 
both sides of a proposed solution. A process should be developed by which a list of 
indicators can be generated. 

• Looking at outcomes as a subset of indicators, not two different evMlies. 

• Re-evaluating the examples of outconrtes and indtoators used to ilkistrate the system. 
Some indicators, such as nutrittonal status, go beyond what school people consider 
their purview and it is difficult to detemiine whether they are indicators or outcomes. 

• Maintaining emphasis on key qualitative variables that do make a difference (e.g., 
school climate) even though they are hard or impossible to nf)easure and clarifying 
some confusion (in the examples) about the distinction made t)etween qualitative and 
quantitative indicators, partk^ularty how qualitative data are treated once aggregated at 
the systems level. Indicators must reflect those system variables that do make a 
difference. 

• Oealing with the limits of assessment and measurement issues related to variance and 
attaching numbers to things when qualitative indk:ators coukl be other than numerical. 

• Examining the various actors and determining who can be held accountable for the 
indk:ators. For what can students (as opposed to the school system) be hekJ 
accountable, for example? 
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• Considering who wili took at the indicators, how often, what inferences will be drawn, 
and what the consequences could/would be in terms of rewards and/or 
training/assistance. 

• Determining and stating up front what purpose the indicators are to fulfill since 
indicators for accountability are different from indicators for formative assessment or 
student evaluation. 



Other issues that were concerned with areas other than indicators included: 

• Dealing with related issues such as cultural variables, (backgrounds and environmental 
staictures within which chiMren live and the adaptatton skills (compensatory skills) an 
indivklual has to make to resporxJ or to demonstrate outcomes. 

• Concem with the interface between the prepared system and outcomes for non- 
special educatton programs and the ability to move within sets of outcomes (e.g., 
college prep vs. vocational) if they are different. 

• Concem about whether interactions between individuals that may or may not have 
anything to do with school shouk) be included in the definitton of educational 
outcomes. 

• Dealing with the responsibility and appropriateness of vartous tong term outcomes 
being klentified as "outcomes of schooling** that suggest school responsibility in 
tracking and measuring. 

• Examining whether the nK)dei itself is too encompassing resulting in a lack of def initton 
or whether the model is too inlellectual rather than practical in whtoh case it may be 
important to shift the focus to a "wortdng* or 'guMing* rather than a conceptual model. 



Sample Comments and Quotations 

**(Your wori<)...has the potential to shape nrujch of the thinking of both special and general 
educatton about how kkls with disabilities fit (or doni) in the refonn efforts that will preoccupy us 
for the next several years." 

**Many calls for higher educational standards purposely exclude special educatton students from 
such requirements or demands. Whether for fear of litigatton or out of benevolence, such 
exclustons will consign special educatton students to the tow skill/tow wage types of jobs. Our 
benevolence will ultimately hurt the students we are trying to protect! The issue is to set 
appropriately high expectattons for each student." 



Consensus: Summary of comments regarding the possibilKy of building 
consensus around the modei of outcomes and system of 
indicators. 



Overview of Comments on Consensus-Buliding 

Most of the twenty respondents to the questton about consensus buikling thought consensus 
was possible (60%). The small proportton who thought consensus wouk) not be attainable (5%) 
dto, however, typtoally acknowledge its importance. An equal number of those who thought 
consensus possible suggested processes by whtoh to achieve it. One respondent emphasized 
that it must be dectoed in advance how much emphasis to place on achieving this goal 
(consensus) and what impact it will have on the project if not achieved. 
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Suggestions for Consensus-Building 

Respondents acknowledged that the NCEO current and proposed strategies for building 
consensus are appropriate including clear conrvnunication with stakeholders by asking for input, 
review, and feedback through typcal channels such as phone, nidil, artk^les and electrons mail. 
Sustaining the consensus building effort was menttoned by several respondents with an 
emphasis on keeping people infonned and updated. An important concern raised was that the 
group be sufficiently broad and include general educators, chief state school offk)ers, state 
directors of special educatton, LEA directors, and university, school board, parent and business 
community representatives. One respondent suggested that representatives from organizations 
and individuals be sorted into levels of involvement such as interested parties, Delphi Group, 
review committee and an NCEO panel. 

Ideas for achieving consensus included: 

• Making the consensus effort sustained as well as connected through periodic 
synthesis summits with other major initiatives irtcluding the Outcomes Project, Joining 
Forces and the NatkHial Goals Panel. 

« Producing a draft for reactton from the fiekl by bringing together a group (e.g., n»30) of 
stakeholders with a broad representatton, maximizing the effort by identifying a 
committed, knowledgeable group to work through the definittons of the outcomes in 
an intensive, face-to-face manner. 

• Using a "think tank*" approach. 

• Selecting deciston makers who have the respect of their peers to engage in open, 
wkJe ranging discussions in all arenas and at all levels about what we are doing and its 
affect on varirus groups. 

• implementing a process where everyone can t>e heard, kleas recognized and 
subjected to a group deciston. 

• 1. Establishing structures and a formal plan for input, debate and resolutton; 2. 
communicating that plan; 3. starting debate; 4. obtaining nominattons from 
organizattons and individuals and sorting them into levels of involvement; 5. having 
everyone serve dual roles of contributing and disseminating information; 6. 
communk^ating regularly and sharing drafts of materials; 7. using formal fiekl review 
(Delphi group); 8. giving the NCEO panel polk^ power to gukle the effort and 9. using 
a small review committee of research experts to valkJate/review wori< at critical 
milestones. 



Comments and Concerns Regarding Consensus 

• Consensus buikiing is not problem solving. It is possible to achieve consensus on an 
issue that can lead to poor resokrttons. Consensus buikiing is most appropriate when 
Issues are clearty defined pro and con. 

• Focus on defining those outcomes upon which we can all agree and not spend too 
much time debating those where there is extensive disagreement. 

• Avoid trying to include everyone, trying for answers to meet all needs/wants, 
maintaining final decision authority (i.e., you need to give up control as final decision 
makers) and working on outcomes and indcators at the same time. 

• Altow for diversity of goals and inr4)lenr)entatk>n by SEAs, LEAs, private schools and 
extant outcomes. Avoid emottonal issues, being realistic and hard headed. 
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Individuals Providing Reactions to Previous Drafts of Working 
Paper 1 



Other NCEO Staff and Consultants 

Bob Algozzine 

MiChal Chvat 
Dean Honetschlager 
Davkl Madson 
Mary Schaefer 

Amy Spiegel 

Charlie Lakin 

State Directors of Special Education 

Linda Hargan, Kentucky 
Mary Beth Fafard, Massachusetts 
Dean Meyers, South Dakota 
Wayne Erickson, Minnesota 
Patrick Campbell, California 
Tom Gillung, Connectk^ut 
Steve Kukk:, Utah 

OSFRS Representatives 

Martha Coutinho 
Lou Danielson 
Martin Kaufman 



Members of the Mid-South RRC Evaluation Forum 

Ken Oisen, MSRRC 
Carol Massanari, MSRRC 
Martha Brooks, Delaware 
Larry Wexler, Distrtot of Columbia 
Nancy LaCount, Kentucky 
David Hayden, Maryland 
Fred Baare, ^k)rth Carolina 
Austin Tuning, Virginia 
Bill Frey, Mtohigan 
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